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Abstract 

There is concern about the potential impacts of climate change on species and 
ecosystems. To address this concern, a large body of literature has developed in 
which these impacts are assessed. In this study, criteria for conducting reliable 
and useful assessments of impacts of future climate are suggested. The major de- 
cisions involve: clearly defining an emissions scenario; selecting a climate model; 
evaluating climate model skill and bias; quantifying General Circulation Model 
(GCM) between-model variability; selecting an ecosystem model and assessing un- 
certainty; properly considering transient versus equilibrium responses; including 
effects of CO2 on plant response; evaluating implications of simplifying assump- 
tions; and considering animal linkage with vegetation. A sample of the literature 
was surveyed in light of these criteria. Many of the studies used climate simu- 
lations that were >10 years old and not representative of best current models. 
Future effects of elevated CO2 on plant drought resistance and productivity were 
generally included in growth model studies but not in niche (habitat suitability) 
studies, causing the latter to forecast greater future adverse impacts. Overly simpli- 
fied spatial representation was frequent and caused the existence of refugia to be 
underestimated. Few studies compared multiple climate simulations and ecosys- 
tem models (including parametric uncertainty), leading to a false impression of 
precision and potentially arbitrary results due to high between-model variance. No 
study assessed climate model retrodictive skill or bias. Overall, most current studies 
fail to meet all of the proposed criteria. Suggestions for improving assessments are 
provided. 



Introduction 

The potential future impacts of climate change are of increas- 
ing concern. Impacts on ecosystems could potentially affect 
the capacity of natural systems to produce wood products, 
crops, livestock, and game. There is also concern about im- 
pacts on biodiversity in general and species extinctions in 
particular. 

To assess these risks, a voluminous literature has appeared 
in which ecosystem or species responses to future possible 
climates are evaluated. These studies often utilize the exper- 
imental literature as a basis for longer term extrapolation. 
Two basic approaches have been used to evaluate long-term 
natural system response (whereas crops can be evaluated ex- 
perimentally and are not considered further here): habitat 
suitability (niche) models and simulations. In habitat suit- 
ability models, the climate space (with or without consider- 
ation of soils, topography, etc.) where a species or vegetation 



type is currently found is characterized by some sort of sta- 
tistical model, and this model is applied to a future climate 
scenario (e.g., Aitken et al. 2008). In simulations, a tree, stand, 
population, or ecosystem growth model is applied to current 
and scenario cases and the results are compared (e.g., Coops 
and Waring 2011). 

While the literature using these approaches is vast, it is 
not systematic or standardized. Many choices must be made 
during the analysis that can lead to large and arbitrary differ- 
ences in study outcomes. For example, Cramer et al. (2001), 
using six dynamic global vegetation models, gave a range of 
outcomes for the global carbon sink in 2 1 00 under rising CO2 
and climate change of 0.3 to 6.6 Pg C y _1 , an immense range 
of 22 x. Verburg et al. (2011) showed that spatial data errors 
in mapped attributes can produce larger uncertainties than 
the process under investigation. Further examples are cited 
in later sections herein. Furthermore, it will be shown herein 
that certain key factors have been left out of many analyses, 
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potentially affecting their interpretation. Such issues can lead 
to the observed widespread failures of model forecasting (e.g., 
Pilkey-Jarvis and Pilkey 2008). 

The purpose of this study, then, is to propose criteria 
for developing ecosystem assessments of response to climate 
change. These criteria involve the choices that must be made 
at each step of the assessment process. In some cases, it is 
sufficient to make explicit the limits that a particular choice 
(e.g., emissions scenario) places on what can be inferred from 
the study. In others, including or excluding a particular factor 
or model can create a particular bias that must be taken into 
account. For example, if a particular climate model predicts 
too much or too little rainfall in the 20th century base runs 
compared to actual rainfall, this will bias or distort results 
of an ecosystem simulation. The presentation of criteria is 
followed by an assessment of the extent to which the criteria 
have been addressed in recent literature. 

Ecosystem Assessment Criteria 

The assessment of any ecosystem impact involves a series of 
steps, especially when long-term effects are being evaluated. 
Failure to exercise care in this process can lead to underesti- 
mation of uncertainty and even to arbitrary results. In order 
to avoid such negative outcomes, a minimal set of criteria 
is proposed for some critical steps in this process. More de- 
tailed criteria could be developed (e.g., for statistical testing), 
but this minimal set is sufficient for the present analysis. The 
criteria for decisions that must be made to put together a 
complete and valid assessment involve the following: 

(1) Clearly define emissions scenario. 

(2) Select climate model(s). 

(3) Evaluate climate model skill and bias. 

(4) Quantify General Circulation Model (GCM) between- 
model variability. 

(5) Select an ecosystem model and assess uncertainty. 

(6) Properly consider transient versus equilibrium re- 
sponses. 

(7) Include effects of CO2 on plant response. 

(8) Evaluate implications of simplifying assumptions. 

(9) Consider animal linkage with vegetation. 

Each of these criteria is discussed more fully in what fol- 
lows. 

Clearly defining an emissions scenario 

It is standard practice to evaluate ecosystem impacts in terms 
of the Intergovernmental Panel on climate change standard 
scenarios for future emissions of greenhouse gases (Intergov- 
ernmental Panel on Climate Change [IPCC] 2007). The first 
factor to consider is that both the scenarios themselves and 
the model ensemble warming due to the scenarios differ be- 
tween the IPCC 2001 (Intergovernmental Panel on Climate 
Change [IPCC] 2001) and 2007 reports. For example, while 
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the A2 2001 scenario is projected to produce 3.8°C warm- 
ing by 2100, the 2007 A2 scenario only projects 3.4°C global 
mean increase by 2100. Thus, studies using the "same" sce- 
nario published at earlier or later dates will not be strictly 
comparable. In addition, model runs from 2001 show wider 
spread than those from the 2007 report. In particular, earlier 
studies are likely to show more impact and should be consid- 
ered to be superseded by more recent ones. Model outputs 
from earlier scenarios and GCMs may still be available but 
should no longer be used and will exaggerate impacts if used. 

A second criterion for utilizing scenarios is to distinguish 
between worst-case scenario and most likely cases. If the 
worst-case scenario is used in the assessment, it should not 
be stated that the simulated impact is "what will" happen, 
but rather is a worst case. If evaluation of a likely scenario is 
desired, the appropriate case should be used. 

Selecting a climate model 

Early (before 1995 or even up to 2000+) GCMs either did 
not simulate precipitation or were known to do so poorly 
(Corte-Real et al. 1995). This includes most of the mod- 
els used in the 2001 IPCC report. For evaluating ecosystem 
impacts, this limitation cannot be addressed by using a con- 
stant precipitation regime. If precipitation is held constant 
and temperature is increased, one obviously would predict 
negative effects on plants. Theory predicts, in contrast, that 
a warmer climate will be accompanied by increased precipi- 
tation (Wentz et al. 2007; Schliep et al. 2010; Stephens et al. 
2010). Whether increased evaporative demand will be bal- 
anced by increased precipitation is an open question and one 
on which different models do not necessarily agree. The fu- 
ture predicted water balance will also vary regionally, from 
wetter to drier to neutral. Thus, it is critical that older GCM 
models/model runs no longer be used for any ecosystem 
studies that involve water balance. Further, in literature sum- 
maries, older studies should not be given equal weight to 
those using more recent GCM results. 

Evaluating climate model skill and bias 

Output from a climate model is often used to evaluate ecosys- 
tem response, perhaps with a regional downscaling first. 
These outputs are virtually always taken at face value for 
conducting impact studies. Any particular model, however, 
may have known skill and bias issues with respect to regional 
climates. Skill can be evaluated by matching the pattern in 
time or distribution of temperatures or rainfall between the 
model and historical data. For example, a GCM might pro- 
duce skillful annual precipitation amounts, but the seasonal 
distribution could be very different from actual. The sea- 
sonal amounts could be critical to properly simulating plant 
growth. Similarly, a GCM could consistently predict temper- 
atures too warm for a region, even though matching historical 
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trends. This could lead to modeled impacts that are not re- 
alistic. In IPCC assessments, in which global trends over 100 
years are being evaluated, these skill and bias issues may not 
affect the calculation of trends, but at the regional/local scale 
for simulating ecosystem response, they cannot be ignored. 

Anagnostopoulos et al. (2010) compared six climate mod- 
els at multiple scales to weather data for the contiguous 
United States. Model results were all warmer than the ob- 
served mean annual temperature and minimum monthly 
temperature by up to 4°C. This bias creates an obvious prob- 
lem for any bioclimatic (niche) or simulation model used to 
forecast future distributions because absolute temperatures, 
not just trends, are critical to biological processes. This is 
particularly true when models are developed using actual 
(not model) climate data, which do not share the model 
bias. Likewise, at the continental scale, they found modeled 
precipitation to be 36% above the true value, which has im- 
plications for any biological model. Schliep et al. (2010) and 
Stephens et al. (2010) noted multiple failings in the ability of 
GCMs to model precipitation, including both bias and lack 
of skill. 

Consider a climate model that produces output at the dry 
end of what occurs in a particular region. A model of forest 
growth is run under constant climate, but because of the cli- 
mate model's temperature bias, the simulated forest is near 
the point at which drought stress would cause dieback. Under 
almost any additional warming, this simulated forest using 
this climate model would show dieback. This bias can be 
addressed by comparing the GCM output to local weather 
data and adjusting the model bias (as Ines and Hansen 2006) 
before simulating forest growth (for both control and impact 
scenarios). If it is determined that skill is lacking (e.g., rainfall 
frequency distribution is wildly wrong or seasonal temper- 
atures are not proportional) then another model should be 
used. In any case, climate model adequacy for the intended 
purpose should be evaluated rather than treating model out- 
put as if it were "data." 

Quantifying GCM between-model variability 

GCMs exhibit considerable variation in outputs (Furrer et al. 
2007), especially at regional scales (e.g., Rowell 2006). For 
example, Woollings (2010) found that simulations for Europe 
differed between models (and from actual weather) for the 
jet stream location, zonal air flow, blocking highs, and other 
key weather patterns. It is important to note that models do 
not differ only in trends or stochastically, but also in terms of 
seasonality and spatial patterns of weather. There may not be 
any sense in which these differences "average out" between 
models. 

Few studies have compared ecosystem impacts using more 
than one climate model. In a study of animal habitat suitabil- 
ity in Spain, Real et al. (2010) found that the differences in 
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predicted habitat between two climate models for an emis- 
sions scenario were greater than the differences between sce- 
narios using the same climate model. Given the large num- 
ber of climate models and their difficulties modeling regional 
climate (noted above), it is apparent that vastly different eco- 
logical impacts could be predicted depending on the model 
used and the geographic region of interest, thereby violat- 
ing criteria of rigor and reproducibility. One way around the 
problem is to use multiple models to sample the range of 
possible projections. This is especially important because at 
the regional scale some models may project wetter and some 
drier conditions. While this solution requires more work to 
perform an analysis, a valid result depends on some estimate 
of uncertainty. 

Selecting an ecosystem model and assessing 
uncertainty 

Just as with climate models, ecosystem models can differ in 
their skill and bias as well as in their suitability for a particular 
task, and may produce widely varying results between models 
(Cramer et al. 2001). The reasons for choosing a particular 
model for an assessment need to be clearly stated. Process 
models can have considerable parametric uncertainty (e.g., 
Larocque et al. 2011) that can carry through the analysis to 
reduce outcome certainty. Ignoring these issues can make it 
look like model output is without error, whereas it is stan- 
dard practice to report confidence intervals on experimental 
results. 

Niche (suitability) models likewise are not guaranteed to 
be accurate (Segurado and Araujo 2004; Araujo and Guisan 
2006). Classification accuracy of niche models does not tell 
the full story. Beale et al. (2008) and Pearson and Dawson 
(2003) provide a more complete discussion of the inherent 
uncertainty in these models including issues of variable co- 
variance and spurious correlation. There is unfortunately no 
modeling method that has unambiguously been shown to be 
universally superior. It is critical to evaluate uncertainty due 
to these issues. 

Properly considering transient versus 
equilibrium responses 

A large part of the ecosystem response literature consists of 
the analysis of habitat suitability (or geographic range) maps. 
Data on climate and other variables are used to predict pres- 
ence of a species, forest type, or ecosystem using regression 
analysis or other tools. This model is then applied to future- 
modeled climates. A large shift in the range is usually con- 
sidered to be an indicator of extinction risk because species, 
especially plants, are not able to migrate rapidly (e.g., Aitken 
et al. 2008; Morin et al. 2008). 

Implicit in these conclusions is the equating of "suitable 
habitat" with the habitat necessary for survival (e.g., Araujo 
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and Guisan 2006). However, the presence of a species means 
that it is competitive in the area where it occurs, not just that it 
can survive there. Small differences in competitive ability will 
eventually work themselves out and show up in distribution 
data, but this process may take a very long time, especially 
for long-lived organisms such as trees. Conditions outside 
the realized niche (i.e., where the species currently occurs) 
cannot be assumed to be lethal without proof. Projected cli- 
mate changes are small compared to even daily temperature 
variation and, for most species, are not near lethal limits 
(e.g., Wertin et al. 2010). This general point was discussed 
in Loehle and LeBlanc (1996) but needs repeating because 
the confusion between transient and equilibrium response 
continues. 

Multiple studies suggest that actual vegetation responses 
to even large climate shifts are slow. Cole (1985, 2009) doc- 
umented a several thousand year transition in the Grand 
Canyon following past warming episodes. Cwynar and Spear 
(1991) showed that boreal forest advance in response to past 
warming took up to 1000 years, with a faster retreat due to 
cooling (which damages trees and prevents regeneration), as 
also shown by Tinner et al. (2007) for boreal dieback during 
the Little Ice Age. Williams et al. (2011) documented a slow 
late Quaternary boreal forest expansion across the North- 
ern Hemisphere. In the boreal forest, where 20th century 
warming would suggest rapid geographic displacement, the 
response has been very slow (Masek 2001; Payette 2007). In 
no case has any study documented a "dead zone" in response 
to a past climate shift, as is implied by static habitat suitability 
models. Simulation models also suggest that such transitions 
should be gradual (Noble 1993; Loehle 2000, 2003). 

Application of these criteria involves the careful delineation 
of implications from a study. It is not necessary that dynamic 
simulations be used. Rather, it is critical that information on 
transient response (relaxation times) can be used to qualify 
conclusions. If the study concerns mobile species, equilib- 
rium might be quickly achieved, but if it concerns trees it 
could take hundreds to thousands of years. Nonoverlap of 
current and hypothetical geographic ranges does not mean 
extinction unless climate lethality can be proven by other 
means. 

Including effects of C0 2 on plant response 

CO2 enrichment directly increases growth and water-use ef- 
ficiency (WUE) in plants (Medlyn et al. 2001; Leuzinger and 
Korner 2007; Loehle 2007; Kohler et al. 2010), with effects 
differing by taxa and CO2 level. Increased WUE is particu- 
larly important when water is limiting, and has been shown to 
mitigate considerable drought stress (e.g., de Graff et al. 2006; 
Wertin et al. 2010). WUE is not considered here in terms of 
hydrology but only with respect to drought response. Free- 
air CO2 exchange (FACE) and open top chambers have both 



shown a positive response of plant growth to CO2 enrich- 
ment (Curtis and Wang 1998; de Graaf et al. 2006). For plant 
growth or habitat suitability models, failure to include CO2 
effects and WUE increases with time over the next 100 years 
will lead to significant overestimates of negative impacts of 
elevated temperature, reduced moisture, or both together 
(Cramer et al. 2001; Pan et al. 2009; Friend 2010). Mechanis- 
tic models that incorporate CO2 effects are generally based 
on experimental results such as FACE experiments, in some 
cases with medium-sized trees over many years. Keenan et al. 
(201 1 ) simulated forest growth with and without CO2 effects. 
Without including CO2 effects in the simulation, all three 
species showed decreased net primary productivity (NPP) 
over time under the warming scenario. However, when CO2 
effects were included, the species exhibited increased growth 
until about 2070 followed by a slight decline, ending at 2100 
with slightly higher NPP than in 2000. How C0 2 effects are 
addressed can make the difference between positive and neg- 
ative growth responses under many scenarios. It is also the 
case, of course, that no CO2 experiments have gone as far as 
100 years, covered large spatial extents, or allowed for species 
shifts, so models incorporating CO2 effects may give unre- 
alistic or exaggerated responses at some time/space scales 
(Leuzinger et al. 2011). 

Evaluating implications of simplifying 
assumptions 

In order to conduct a geographic analysis of climate effects, 
it is often necessary to make simplifying assumptions due 
to data resolution. We may, for example, have GCM output 
at 1° x 1° resolution. In the real world, however, fine-scale 
topographic effects moderate regional climate in complex 
terrain. An analysis based on coarse-scale data might suggest 
that no suitable habitat remains under some scenario, when 
in fact topography might provide for refugia (Scherrer and 
Korner 2010; Dobrowski 2011; Godfree et al. 2011). Other 
simplifications might be made, such as using a single very dry 
soil type for the simulation (e.g., Coops and Waring 2011). 
The criterion here is that such simplifications, while perhaps 
necessary because of data or computational limitations, have 
implications that qualify the results and should be discussed. 
In addition, geographic data can have errors resulting from 
vegetation classification, measurement error and bias, ag- 
gregation error, and other sources, leading to effects possibly 
larger than those resulting from the scenarios being evaluated 
(Verburg et al. 2011). 

Considering animal linkage with vegetation 

When suitability/range map-type models are developed for 
plants, there are good reasons to suppose that climate directly 
determines plant distribution (as modified by competition 
and fire, or course). It is equally easy to develop models for 
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animals' range, but these models might actually be proxies for 
the vegetation on which the animals depend as much as for 
climate per se. That is, spurious correlation is a real danger 
in the absence of experimental data. For example, ruminants 
depend on grasslands and waterfowl depend on marshes and 
lakes. If an animal depends on certain vegetation that exhibits 
a long lag in responding to climate, the animal may likewise 
persist in its current range unless its physiological tolerance 
is actually exceeded. If the niche model uses climate vari- 
ables, the projection of animal response could be unrealistic. 
Thus, it should be recognized that suitability models are cor- 
relational rather than fundamental, and investigators should 
carefully consider model output in this light. 

Literature Survey 

A literature survey was conducted to evaluate compliance 
with the assessment criteria suggested here. All issues of 
Global Change Biology in 2009 and 2010 and the first four 
issues in 201 1 were assessed. All terrestrial studies discussing 
future impacts on species or ecosystems were considered (not 
carbon or fire studies). Experimental and field studies were 
outside the scope of the review. A total of 20 papers were 
evaluated (Table 1), with one study evaluated under two cat- 
egories. In the author's experience, these 20 papers are similar 
to others in the field and serve to illustrate the points made. 

Growth models 

Nine papers (seven of vegetation) with some sort of dynamic 
or growth model utilized a variety of approaches and often 



met most of the criteria proposed for impact assessments. 
Most used the most recent IPCC AR4 climate simulations. 
Keenan et al. (2011) did underestimate outcome variability 
by only using one climate simulation and one forest growth 
model, but is the only study that directly compared niche 
and growth model results (using multiple niche models but 
a single vegetation model). The inclusion of CO2 effects led 
to an opposite conclusion on impacts (positive) compared to 
leaving them out (negative). Doherty et al. (2010) evaluated 
effects of future climate on NPP of East Africa. They used out- 
put from nine GCM models to capture uncertainty but only 
a single vegetation model that did simulate CO2 enrichment 
effects. All cases projected increased NPP in the study area. 
Poulter et al. (2010) simulated response of the Amazon forest 
to CO2 and eight different GCMs as well as vegetation model 
parameter uncertainty. They found that between-climate- 
model differences dominated outcome spread compared to 
CO2 or vegetation model parameter uncertainty, with future 
precipitation uncertainty the greatest unknown determinant 
of outcomes. 

Chen et al. (2010) evaluated response of Douglas-fir to 
projected climate change using a tree ring index model of 
growth response. Variability was captured by using output 
from five climate models. The inability of this approach to 
include effects of rising CO2 is problematic because the main 
impacts of climate change in their analysis were manifested 
via drought stress, which rising CO2 and thus increasing 
WUE would help ameliorate. Xu et al. (2009) simulated forest 
growth in northern Minnesota using a single process model. 
CO2 was included in the model. Future climate data (27 



Table 1. Percent compliance with criteria for literature survey cases (percentages calculated only for criteria relevant to the studies in question). NA 
reflects that criterion not relevant to that model type. 
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0.57 
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0.6 
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0.4 
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0.6 



NA = not applicable 
a criteria areas follows: 

(1) Clearly define emissions scenario. 

(2) Select climate model(s). 

(3) Evaluate climate model skill and bias. 

(4) Quantify GCM between-model variability. 

(5) Select an ecosystem model and assess uncertainty. 

(6) Properly consider transient versus equilibrium responses. 

(7) Include effects of CO2 on plant response. 

(8) Evaluate implications of simplifying assumptions. 

(9) Consider animal linkage with vegetation. 
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profiles) were mostly from older GCMs used in the third IPCC 
assessment in 200 1 . The large number of climate runs allowed 
the large uncertainty due to forecasting to be quantified. 
Morin et al. (2009) used a process-based statistical model 
for leaf unfolding for 22 North American tree species as a 
function of climate. A single GCM run from 200 1 was used for 
the A2 and B2 scenarios only. The effect of CO2 on phenology 
was not assessed. Spatial resolution was probably not an issue. 
With a single older GCM run, there is no way to evaluate the 
representativeness of their forecasts. Scheiter and Higgens 
(2009) used a mechanistic model of grassland and forest 
to evaluate response of the vegetation of Africa to climate 
change. The model included CO2 effects that were believed 
to account for increased forest area and biomass as well as 
the increase in total vegetated land (at the expense of desert) 
by 2100. Spatial resolution was good at the scale of 1 ha. The 
GCM used was a 2007 model run, but the lack of intermodel 
comparison makes it difficult to evaluate results. 

Two animal process models were found. Gilg et al. (2009) 
explored effects of future climate on Arctic predator-prey 
systems. The climate scenarios were qualitative and hence 
difficult to evaluate. Snail et al. (2009) used seven up-to-date 
GCM models and a process model for plague in prairie dogs. 
By sampling from the plague model parameter uncertainty 
space as well as the multiple GCMs they were able to do an 
exemplary job quantifying uncertainty, although they did not 
assess GCM bias or skill. 

Overall, process-based studies did a good job assessing un- 
certainty due to between climate model variation and within 
ecosystem model parametric uncertainty, although only Snail 
et al. (2009) and Poulter et al. (2010) evaluated both. Most 
used current climate models but none evaluated GCM skill 
and bias issues. 

Niche models 

All seven studies using niche models for plant distributions 
(Bradley 2009; Bradley et al. 2009; Randin et al. 2009; Fee- 
ley and Silman 2010; Dirnbock and Rabitsch 2011; Dlamini 
2011; Keenan et al. 2011) failed to consider CO2 effects on 
future growth. Because much of the impact of future climate 
on plants results from net water deficiency, the lack of consid- 
eration of CO2 effects calls results into question, especially in 
light of Keenan et al.'s (2011) results discussed above. Use of 
future climate scenarios was mixed. Keenan et al. (201 1) used 
IPCC AR4 climate model results, but Dlamini (2011) used a 
result from 2000, which is rather old. Dirnbock and Rabitsch 
(2011) and Feeley and Silman (2010) used simple increases 
in temperature, including an extreme 8°C rise in the latter 
case, with no increase in precipitation. Failure to consider fu- 
ture precipitation changes generally will make impacts more 
negative for plants. Uncertainty due to niche model differ- 
ences was considered by Keenan et al. (20 1 1 ) , and that due to 



between-climate model differences by Dlamini (2011). The 
equilibrium assumption (equating niche model changes with 
immediate population changes) was universal in these stud- 
ies, with only a mention of possible lags by Dirnbock and 
Rabitsch (2011), although possible lags were considered to 
be in terms of mere decades. Two studies of invasive plants 
(Bradley 2009; Bradley etal. 2009) using niche models used 10 
AR4 GCM runs and were thus able to quantify model-based 
uncertainty, but did not assess model skill or bias and did 
not clearly cite the model runs used. Because they only eval- 
uated potential habitat for invasion, there was less problem 
with the equilibrium assumption, but they did assume that 
future unsuitable habitat for the invasive species would im- 
mediately become a conservation opportunity. Randin et al. 
(2009) compared scenarios at 50 km x 50 km versus 25 m x 
25 m plot scales. Because they interpolated climate to ac- 
count for topography at the smaller plot size, the local-scale 
models showed persistence of up to 100% of species that the 
European -scale model predicted would lose their entire habi- 
tat. The study was limited by the use of (apparently) an older 
single (approximately 2000) GCM run. The net effect in all 
these models of excluding the beneficial effects of elevated 
CO2 and assuming rapid vegetation changes due to changes 
in available suitable habitat is to greatly amplify likely neg- 
ative effects or even to convert positive effects into negative 
ones, as in Keenan et al. (201 1). Likewise, niche models at too 
coarse a resolution will miss refugia from climate change. 

Five niche models were used to assess animal distributions 
(Jarema et al. 2009; Carroll et al. 2010; Carvalho et al. 2010; 
Rebelo et al. 2010; Habel et al. 2011). These studies used 
climate projections from 2004 or 2005 except for Carvalho 
et al. (2010) for which the date of the climate model could 
not be determined and Jarema et al. (2009), which used a 
2001 GCM run. Carvalho et al. (2010) used more than one 
niche model (nine) and GCM (three) and Jarema et al. (2009) 
used two GCM models and multiple niche models. Carroll 
et al. (2010) developed animal niche models that included 
vegetation and held vegetation constant based on considera- 
tion of lags that were likely to occur in forests of the Pacific 
Northwest United States. Jarema et al. (2009) found niche 
models based on habitat to be almost as good as those using 
climate variables alone. All other studies used niche models 
dominated by climate variables and thus ignored the possibil- 
ity that animals may be found in their current habitat based 
on the vegetation, which would probably respond to climate 
change in a lagged fashion. They all further assumed equi- 
librium response, whereas animal distributions are likely to 
be heavily influenced by competition and predation effects 
that would take time to come to equilibrium in a new cli- 
mate. Finally, a fairly extreme scenario of 6°C warming was 
used by Rebelo et al. (2010). Simplifying assumptions that 
may have affected results include use of 35 km 2 grid cells in 
Dirnbock and Rabitsch (2011), only a simple temperature 



68 



© 201 1 The Author. Published by Blackwell Publishing Ltd. 



C. Loehle 

gradient (no heterogeneity or precipitation) in Feeley and 
Silman (2010), the coarse spatial scale used in Carvalho et al. 
(2010) that would not allow for refugia, and the assumption 
in Rebelo et al. (2010) that bats are dispersal-limited. Over- 
all, uncertainty was underestimated and effects were probably 
exaggerated in this set of studies. 

Discussion 

The criteria suggested in this study are intended to guide de- 
cisions in assessments of future climate change impacts and 
to help prevent results from being arbitrary or showing false 
precision due to lack of uncertainty information. The litera- 
ture survey revealed that recent studies have not considered 
some of the proposed criteria (Table 1). Violation of any of 
the criteria can be serious, but some are more easily reme- 
died than others. The criteria are next considered in turn, 
with suggestions for improved assessments. 

The examined papers generally referred back to IPCC AR4 
emissions scenarios (IPCC 2007). Usually, either the A2 sce- 
nario was used or a range of scenarios was evaluated. This 
standard set of scenarios helps with interstudy comparisons. 
Some studies used arbitrary temperature increases rather 
than model outputs, which has the added problem that pre- 
cipitation was held constant or ignored. If simple temperature 
increases are evaluated, they should at least be more clearly 
related to model scenarios. It is difficult to justify leaving 
precipitation changes out of an assessment and not diffi- 
cult to add this factor. Recent generation climate simulations 
were not necessarily used, although the growth model stud- 
ies mostly used up-to-date results. Four of seven plant niche 
models used out-of-date climate models. There is really no 
good reason to use old climate simulations. 

No study took into account the skill or bias of the climate 
models. This is problematic, especially because many used 
models for current ecosystems calibrated with actual weather 
data and compared them to future simulated climates in 
which a bias (offset) could exist. At least a qualitative assess- 
ment of climate model outputs compared to the study area 
seems necessary before doing an assessment. Of the sources 
of uncertainty (climate model, and niche model or growth 
model), only three studies (Jarema et al. 2009; Carvalho et al. 
2010; Poulter et al. 2010) considered both and many stud- 
ies included only a single model and GCM dataset. It is not 
impossible to avoid this problem by explicitly using the mul- 
tiple climate model outputs, which are becoming increasingly 
available for regional scales. Likewise, ensemble means of cli- 
mate data might be available and ensemble means or overlap 
for niche models (Araujo and New 2007) are not difficult to 
develop, since the statistical tools are widely available. 

Studies using niche models mostly failed to properly con- 
sider lags in vegetation response to climate change. They as- 
sumed that habitat defined as "unsuitable" by the niche model 
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was immediately uninhabitable by the species or vegetation 
without presenting evidence that the altered climate would 
be actually lethal. Instead, competitive displacement, an in- 
herently slow process that may take centuries (Noble 1993; 
Loehle 2000, 2003), could eventually result in the changes 
suggested by niche models. While niche models are inherently 
equilibrium based, it is possible to perform some assessment 
of nonequilibrium response (e.g., Araujo and Pearson 2005). 
In addition, data do exist on drought and heat tolerance of 
many species, and such data should be consulted before mak- 
ing claims that species or ecosystems will perish/vanish over 
the next few decades due to a few degrees warming. If the 
temperature rise is within the thermal tolerance of a species, 
then range shifts, not extinctions, are going to result and 
may take centuries. Consulting the literature to evaluate the 
likely time dynamics of the output of niche models is not a 
burdensome requirement. 

Most studies using growth models included beneficial ef- 
fects of elevated CO2 on plant growth. It is noteworthy that 
these models forecast much less impact or even positive ef- 
fects compared to niche models, which are inherently unable 
to account for future elevated CO2 effects. While the ex- 
act effects of elevated CO2 over the long term are not known 
(e.g., Leuzinger et al. 201 1), the experimental literature seems 
to support a growth-enhancement conclusion and models 
have widely adopted this result. It would seem that dynamic 
(mechanistic) models of plant response to climate change 
should be preferred, though niche models are much more 
widely published, perhaps due to simplicity and cost. Cost 
alone does not justify a method that leads to the wrong an- 
swer. At the very least, the result of any niche analysis needs 
to be qualified by discussion of how CO2 effects would alter 
the result. 

The primary simplifying assumption that impacted results 
in a number of studies was the reduction of spatial hetero- 
geneity considered, either due to grid scale or to elevational 
gradient representation (e.g., representing it as a smooth 
curve). Spatial heterogeneity can provide refugia, and fail- 
ure to consider this effect will tend to exaggerate impacts, as 
clearly shown by Randin et al. (2009). The limiting factors 
for performing more spatially resolved analyses are com- 
puter time and input data. The former can be overcome by 
distributed computing or weekend runs. The latter is at least 
mitigated by using the highest resolution data available rather 
than aggregated data. The conceptual and analysis steps are 
identical in either case. 

Finally, in animal impact studies, a pervasive trend was 
to model animal response solely as a function of climate. If 
animals select habitat based on vegetation and the response 
of vegetation to changing climate is lagged, the response of 
animals will probably also be lagged. The assumption that 
animals are limited by climate as defined by a niche model is 
virtually unverified. 
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When study results can be completely altered by including 
or excluding a factor such as CO2 or spatial heterogeneity and 
when between-model forecasts can be extremely variable, it 
is essential that these and other known factors be recognized 
and addressed in any analysis of climate change effects. The 
majority of surveyed studies might have yielded very different 
results if the criteria suggested here had been applied. More 
attention to criteria such as those proposed would lead to 
much more reliable and useful impact assessments. It has 
been shown above that most of these problems are easily fixed, 
require a modest effort (e.g., using niche model ensembles), 
or can be dealt with in the discussion of results. 
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